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“A group of people who work together” 
— Merriam-Webster 
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¢ Members share a common goal 
¢ Group needs outweigh individuals 
¢ Must have common ground & trust 


AFM eC »~rMAIAL AAaVWIRN e \} 
Norms Srning naviors) 


¢ Background (experience, training, 
knowledge, culture, etc.) 

¢ Org structure (chain of command) 

¢ Work protocol (doctrine) 


¢ Communication 
¢ Coordination 
¢ Collaboration 
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Signals 
¢ Limited content (few bits) 
¢ Convey awareness, intent, state, etc. 
¢ Numerous mechanisms 
(combine for emphasis & redundancy) 
= Auditory 
= Gaze 
= Gesture 
= Motion 


Language 
¢ Extensive content (many bits) 
¢ Convey high level of detail 
¢ Specific vs. general 
= Task specific 
= Domain specific 
= Natural 
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“Harmonious functioning” 


¢ Making sure that two or more people (or groups of people) can work 
together properly and well 


¢ Involves integration of activities, responsibilities, etc. to ensure that 
resources are used efficiently and effectively 


¢ Requires control, organization, monitoring, etc. 


Effective coordination requires: 

¢ Common ground: mutual 
knowledge that supports 
joint activity 

¢ Directability: assessing and 
modifying individual actions 
within joint activity 

¢ Interpredictability: being able 
to predict what others will do 
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¢ Multiple individuals working together to achieve a shared objective 
¢ Requires communication and coordination 
¢ Involves sharing of knowledge, intention, and goals 


Collaborative tasks 
¢ Tightly coupled: each participant depends on the actions of other 
individuals (jointly pushing a sofa) 


¢ Loosely coupled: each participant engages in complementary actions 
towards a shared goal (splitting up to search) 
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Assumptions 
¢ Robots should be team members 
¢ Robots can be successful and trusted team members 
¢ Human teams are a good model for human-robot teams 


Robots have (engineered) limits 
¢ Robots often cannot handle anomalies, edge cases, & corner cases 
¢ Appearance can be deceiving: a humanoid robot # a human 


Humans have difficulty creating mental models of robots 
¢ Hard to set and manage expectations of robot behavior & performance 
¢ Teamwork may be unnatural and inefficient (high human workload) 


Robots have difficulty recognizing human intent 
¢ Robot may not act at the right time or respond properly 
¢ Teamwork may be slow and jittery 
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¢ “Robot as tool” is only one model 
¢ Not just co-located or line-of-sight 
> Peer-to-peer teaming is also important 


¢ Human-robot interaction is still slow and 
mismatched (compared to human teams) 

¢ Easy for robots to impede the human 

> Loosely-coupled teaming is essential 


Distributed teams 
¢ Require coordination and info exchange 


¢ Require understanding of (and planning for) 
each teammate’s capabilities 


> Effective protocols and tools are critical 
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Research @ NASA Ames 


Part 1: Communication 
¢ Signaling for non-humanoid robots 


¢ Convey robot state and intent using 
dynamic light and sound 


¢ Ambient and active communication 
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Situation awareness 
¢ Robot is positioned out of the human’s view 
¢ Signals can indicate the presence and location of the robot to facilitate 
SA (at multiple levels) 
¢ Signals can facilitate prediction and planning (avoid conflict before it 
occurs, avoid dangerous situation, etc). 
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Spatial negotiation 


¢ When humans and robots must co-exist in the same space, there is often 
a need for spatial negotiation 


¢ Cannot always rely on pre-defined rules (e.g., “right of way”) due to 
ambiguity and uncertainty 


¢ Signaling (lights, movement, sound, etc) is an effective manner to 
communicate intent and elicit action. 
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Considerations 
¢ What to convey (importance of the information) 
¢ When to convey (timing of the information) 
¢ How to convey (constrained/modulated by configuration, situation, etc..) 
¢ To whom do we convey (user role, capability to receive/respond, etc.) 


SIGNAL 


( 


CRITICALITY INFORMATION CONTENT 
ATTENTION RESPONSE TYPE CAPACITY 


°° * (A 
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What to convey’? 


Robot states 
¢ Condition 
= Operational status: health, control mode, faults 
¢ Knowledge 
= Information the robot possesses about itself, the task, and the world 
¢ Activity 
= Actions the robot is taking (or attempting) to take — often task related 
¢ Affect 
= The “emotional state” of the robot 
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Signal design 
¢ Use Case Analysis 
= Describe the robot’s goals using use case descriptions 
¢ Communication Analysis 
= Describe the robot’s communications within each use case 
¢ Failure Analysis 
= Identify the risks of a communication case not occurring 
¢ Priority Ranking 
= Weighting different types of risk (e.g., inefficiency vs. human injury) 


Risk: Human Injury 


5} 5 Critical 
High 
Cs Moderate 
24 Low 
$ Very Low 
le =e = 2 
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Probability 


E. Cha, Y. Kim, T. Fong, and M. Mataric (2017) “A system for 


designing human-robot communication” (in submission) 
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High Demand 
awareness Reaction 


Interrupt 


Low 
awareness 
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Interrupt until human responds / intervenes 


Request attention from human 


Help humans decide their further action 


Help humans monitor robot's overall action 


Optional (non-critical) information 
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Considerations 
¢ Embodiment 
= Form: How does the robot’s physical form affect signaling capabilities? 
= Generalizability: How can the same signals be utilized across platforms? 
¢ Signal design 


« Intuitiveness: How to utilize non-humanoid communication modalities to 
signal in an intuitive manner? 


= Complexity: How to create signals of varying complexity utilizing non- 
humanoid communication modalities? 


¢ External factors 


= Environment: How to account for the environment (e.g., perceptual 
conditions, ambient noise) and external events in signaling? 


¢ Psychological factors 
= Perception: How to control humans’ perceptions of the robot’s signals? 
= Evaluation: How to accurately evaluate signals in real world scenarios? 
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¢ Free flying robot inside the Space Station 
¢ All electric with fan-based propulsion 
¢ Three smartphone computers 
¢ Expansion port for new payloads 
¢ Open-source software 
¢ 30x30x30 cm, 8 kg 
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¢ Mobile sensor 
¢ Remotely operated camera 
¢ Zero-G robotic research Computers 
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Autonomy 

¢ Docking & recharge 

¢ Perching on handrails 
¢ Vision-based navigation Signal lights 


NASA 
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Astrobee on the Space Station (concept) 
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Situation | 


Perching 


Motion 


On/Off 
state 


Perching 
progress 


Camera 


mode 


Low power Stuck 
Goal 
(research 
plan / 
camera 
mode / 
search 
mode) 


Action or 
task 


Moving 
direction to 
warn 
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Destination 


States 


pointing 
where to 
move - 
heading 
(handle) 


Progress 
(doing/ 
completing 
/ awaiting 
further 
order) 


Priority / 
urgency 


Speed or 
accel. 


Purpose 


Assistance 

required for 
task or fault 
recovery 


Trajectory Coming 
into view 


Adjacency 
(to human 
or obstacle 


19 


ovihitersiilelamls\\csi ts 


Ambient State Alert State 


Ignore _ Change Blind = MakeAware Interrupt} = Demand Action | 


Need help state ¢—_> 
(Blocked robot view / Low battely / Stuck by obstacles) 


vovng sate T° 


Instant state | | Alert state | 
(Changing direction) ‘(Running into crew} 


Less active state 


(Stationary me Perched / Hibernated /Docked) 


Progressive te : : 


(Battery charging / Data transmitting / Task executing). 


®— Start state <— Transfer state @— End state 
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Possible signals 


Physical Distance 


Separate (invisible) Far (ed [eox-y:) 

; Alerting+ ] 

SOUND L Alerting a state voce commana 
MOVEMENT Motion Orientation, Gesture 

LIGHT | Conveying intent | 


Conveying ] 
__intent/information _ 


(= 
TOUCH SCREEN Conveying intent 


— 


Receiving | 
TOUCH —_fuman input ; 


© Human-robot teaming 21 


Mifolalmcjielar<lilare mie) migstouri\s|aremgelele) ts 


beacon blinker 


thruster 


D. Szafir, B. Mutlu, and T. Fong (2015) “Communicating 


directionality in flying robots”. ACM/IEEE HRI Conf. 
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Research @ NASA Ames 


Part 2: Coordination 
¢ Achieve common (joint) objective 
¢ Independent human and robot activities 


¢ Robots work before, in parallel (loosely 
coupled) and after humans 
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i | Jack Schmitt & Lunar Roving Vehicle 
soe 17 (1972) 
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What's changed since Apollo? 
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Robots before crew 
¢ Prepare for subsequent human mission 
¢ Scouting, prospecting, etc. 


¢ Site preparation, equipment deployment, 
infrastructure setup, etc. 


Robots supporting crew 
¢ Parallel activities and real-time support 
¢ Inspection, mobile camera, etc. 
¢ Heavy transport & mobility 


Robots after crew 
¢ Perform work following human mission 
¢ Follow-up and “caretaking” work 
¢ Close-out tasks, maintenance, etc. 
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Robotic Recon Project 


Objectives 
¢ Assess value of robotic recon 


¢ Study coordinated human-robot 
field exploration 


¢ Fold lessons learned into lunar 
surface science ops concepts 


Results 


¢ Captured requirements (instruments, 
comm, nav, etc.) for robotic recon 

¢ Assessed impact of robotic recon on 
traverse planning & crew productivity 

¢ Learned how to improve human 
productivity & science return 


| Space Exploration Veh ay 


M. Bualat et al. (2011) “Robotic recon for human exploration: 


i>, method, assessment, and lessons learned”. GSA special paper 483. 
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Field experiment (2009) 


Pre-Recon 


Mar 1 — June 1 June 14 — 26 July 1 — Aug 15 Aug 29 — Sep 3 
¢ Satellite images ¢ K10 at BPLF ¢ Recon images ¢ SEV at Black Point 
* Geologic map ¢ Ground control ¢ Terrain models ¢ Science backroom 


i at NASA Ames at Black Point 
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Black Point Lava Flow 
¢ 65 km N of Flagstaff, AZ 


¢ Analog of the “Straight Wall” 
(Mare Nubrium / Rupes Recta) 


¢ Basaltic volcanic rocks & 
unit contacts 
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Robotic recon results 


“West” region 


¢ Pre-recon traverse plan was Vg sae igs 
designed to be cil yy x. 
- Rapid area coverage i Di 
(visit 5 hypothesized NNER Pb » 
geologic units) a . a. & ie! : Sl Nd 20A_06 PML 
= Single visit / sortie ; aaa Fis ane 
¢ Post-recon traverse plan is 
different 
= More flexible & adaptable 
= Recon data supports 
by crew 


¢ Impact of recon EN hes re ~ 


« Reduced science uncertainty VE OORE + SN ae 
= Improved target prioritization } : 


Post-recon 


T. Fong et al. (2010) “Assessment of robotic recon for 
human exploration of the Moon”. Acta Astronautica 67 (9-10) 
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An exploration problem 
¢ Never enough time for field work 
¢ “If only | could have...” 
= More observations 
= Additional sampling 
= Complementary & supplementary work 


The solution 


¢ Use robots to “follow-up” after 
human mission is completed 


¢ Augment human field work with 
additional robot activity 


¢ Use robots for work that is tedious 
or unproductive for humans 
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Arctic Ocean 


Devon 
Island 


Haughton Crater 
¢ 20 km diameter impact structure 
¢ ~39 million years ago (Late Eocene) 
¢ Devon Island: 66,800 sq. km (largest uninhabited island on Earth) 
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Shackleton Crater ‘Haughton Crater 
2005 Arecibo radar image radar image 


¢ Polar impact structures: mixed impact rocks & ejecta blocks 
¢ Subsurface water ice 
¢ Remote, isolated, difficult to access 
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Crew mission (July 2009) 
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_ and Pascal Lee = tetera oom = : and Pascal Lee 
Geologic Mapping Geophysical Survey 
¢ Document geologic history, ¢ Examine subsurface structure 
structural geometry & major units ¢ 3D distribution of buried ground 
¢ Example impact breccia & clasts ice in permafrost layer 
¢ Take photos & collect samples ¢ Ground-penetrating radar: 


manual deploy, 400/900 MHz 
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Stratified "> © 4 contact between 
sediments [7 & =| carbonates 


View East ) Gray carbonate 
into crater aw : ~ )} breccia 
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eophysical survey results 
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Robotic follow-up results 


Explanation 
LJ Alluvium 
Glacial Deposits 
() intracrater, Stratified Sediments 
(7) Grey Carbonate Breccia 
Carbonate w/o stratal continuity 


Geologic Mapping 
¢ Verified the geologic map in ye é 
multiple locations Sse SR |S satctconr 
¢ Amended the geologic map in | BOY. ES “teste 
multiple locations 
¢ In some places, robot data was 
ambiguous, or lacked sufficient 
detail to re-interpret the map 
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Geophysical Survey 
¢ Enabled study (correlation of 
surface & subsurface features) 
of terrain “polygons” 
¢ Determined average depth of 
subsurface ice layer and features 
(ice wedges) 


T. Fong, M. Bualat, et al. (2010) “Robotic follow-up 
ise for human exploration”. AIAA Space Conf. 
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Research @ NASA Ames 


Part 3: Collaboration 
¢ Humans support autonomous robots 


¢ Focus on cognitive tasks (planning, 
decision making, etc) 


¢ Human-robot team may be distributed 
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Our focus 
¢ Study how humans can remotely support robots 


¢ Address the many anomalies, corner cases, and edge cases that 
require unique solutions, which are not currently practical to develop, 
test, and validate under real-world conditions 

¢ Humans provide high-level guidance (not low-level control) to assist 
when autonomy is inadequate, untrusted, etc. 


asa Human-robot teaming 47 


Human-Robotic Partnership (p. 22) 


Tele-Presence 


Tele-presence can be defined as tele-operation of a robotic 


asset on a planetary surface by a person who is relatively 


close to the planetary surface, perhaps orbiting in a space- 


craft or positioned at a suitable Lagrange point. Tele-presence 


is a capability which could significantly enhance the ability 


of humans and robots to explore together, where the specific 


exploration tasks would benefit from this capability. These 


tasks could be characterized by: 


High-speed mobility 

Short mission durations 

Focused or dexterous tasks with short-time decision-making 
Reduced autonomy or redundancy on the surface asset 
Contingency modes/failure analysis through crew interaction 
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ISECG 


International Space Exploration 
Coordination Group 
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Key Points 


¢ Demo crew-control surface telerobotics 
(planetary rover) from ISS 


¢ Test human-robot conops for 
future exploration mission 


¢ Obtain baseline engineering data 
(robot, crew, data comm, task, etc) 


Implementation 
¢ Lunar libration mission simulation 


¢ Astronaut on ISS (in USOS) 
¢ K10 rover in NASA Ames Roverscape 


ISS Testing (Expedition 36) 
June 17, 2013 — C. Cassidy, survey 
July 26, 2013 — L. Parmitano, deploy 
Aug 20, 2013 — K. Nyberg, inspect 
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Orion MP 
¢ 60,000 km nevone lunar pide: 
¢ Allows station keeping with minimal fuel 
¢ Crew remotely operates robot 
¢ Does not require human-rated lander 


Human-rol 
¢ Crew emoialy scores surface robot 
from inside flight vehicle 


Crew works in shirt-sleeve environment 
¢ Multiple robot control modes 


Moon’s 
Depart Free-Return | Orbit 


Earth 7 Trajectory o 
ay L, . ° L, 


Credit: (Lockheed Martin / LUNAR) 


Surface 
Mission Start 


Surface { 
Mission End 


Orion Orbit 
Insertion , 


50 


ISS test setup 


“Live” Rover 
ASY=Vak-xo) mr: Tale| 
Instrument 


Data 
(telemetry) 


lh _— 


Rover/ 
Science 
Data (e.g. 
imagery) 


Teysuesy all 1S9}-}S0q 


nterface 
Instrumentation & 
Evaluation Data 


Rover Plan 
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K10 rover at NASA Ames 
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Chris Cassidy uses the “Surface Telerobotics Workbench” 
to remotely operate K10 from the ISS 
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Crew Session #2 — K10 deploying simulated polymide antenna (2013-07-26) 
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ISS Mission Control (MCC-H) during Surface Telerobotics test 


View of robot interface and K10 at ARC 
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Crew control of K-10 rover 


an 
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Deployed simulated polymide antenna (three “arms”) 
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Crew Session #3 — Karen Nyberg remotely operates K10 (2013-08-20) 
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K10 documenting simulated polymide antenna 


Assessment approach 


Metrics 


Mission Success: % task sequences: completed normally, ended abnormally 
or not attempted; % task sequences scheduled vs. unscheduled 


Robot Utilization: % time robot spent on different types of tasks; comparison 
of actual to expected utilization 


Task Success: % completed normally, ended abnormally or not attempted; 
% that ended abnormally vs. unscheduled task sequences 


Contingencies: Mean Time To Intervene, Mean Time Between Interventions 
Robot Performance: expected vs. actual execution time on tasks 


Data Collection 


automatic 


Data Communication: direction (up/down), message type, total volume, etc. 
Robot Telemetry: position, orientation, power, health, instrument state, etc. 
User Interfaces: mode changes, data input, access to reference data, etc. 
Robot Operations: start, end, duration of planning, monitoring, and analysis 
Crew Questionnaires: workload (Bedford Scale), situation awareness (SAGAT) 


M. Bualat, D. Schreckenghost, et al. (2014) “Results from testing crew-controlled 
surface telerobotics on the International Space Station”. 12" I-SAIRAS 
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Productivity 


¢ Productive Time (PT) = astronaut and robot performing tasks 
contributing to mission objectives 


¢ Overhead Time (OT) = astronaut and robot are waiting 
¢ Work Efficiency Index (WEl) = Productive Time / Overhead Time 


Productivity Total Phase Time PT OT %PT %OT WEI 


Highly productive 
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Publi ships 
. Gesuic God. 15): collaborative 
testing of sensors and vehicles 


¢ Nissan (2014-17): cooperative 
software development 


IC; /pr riV 


. Eveand ndaledae of commercial 
autonomous systems 


¢ Develop protocols and best practices 
for safe testing of real-world autonomy 


¢ Transfer NASA technology to 
terrestrial ane 

ech INO ol ¢ 

¢ Safe es igain in cian environment 


¢ Leverage NASA expertise in 
autonomy, robotics, safety critical C nn 
eS systems, and vehicle systems : eC 
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When a construction worker uses hand gestures to provide guidance, or 
direction, no autonomous car today can reliably make the right decision. 


¢ When the sun is immediately behind a traffic light, most cameras will not 
be able to recognize the color of the signal through the glare. 


¢ If we see children distracted by the ice cream truck across the street, 
we know to slow down, as they may dash toward it. 


— Andrew Ng (Wired, 3/15/2016) 
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Vehicle assist: Situation assessment 
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Vehicle assist: High-level guidance 
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CES 2017 demo 
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Communication 


¢ Design appropriate signals (compact, legible, etc) to convey 
robot intent, status, etc. 


¢ Signals may need to vary based on distance, environment, situation, etc. 
¢ Do not need natural language to be effective 


Coordination 
¢ Must make it easy for humans to work with robot (and vice versa) 
¢ Human-robot teaming is not just side-by-side, closely coupled actions 


¢ Consider how robots working before, in support, and after humans can 
be effective at achieving a goal 


Collaboration 
¢ Identifying and building upon interdependence is essential 


¢ Not all tasks can be planned in advance -- teaming must support 
spontaneous actions 


¢ An effective team works together to achieve a shared objective 
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Intelligent Robotics Group 
Intelligent Systems Division 
NASA Ames Research Center 


irg.arc.nasa.gov 
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